Method and system for multi-wavelength depth encoding for three-dimensional range geometry compression

ABSTRACT

A method for generating encoded depth data includes receiving digital fringe projection (DFP) data corresponding to a three-dimensional structure of a physical object, and generating first and second fringe encodings for a first predetermined wavelength based on the DFP data at a first coordinate. The method further includes generating third and fourth fringe encodings for a second predetermined wavelength based on the DFP data at the first coordinate, the second wavelength being longer than the first wavelength, and generating a combined fringe encoding based on the third fringe encoding and the fourth fringe encoding. The method further includes storing the first, second, and combined fringe encoding data in a pixel of two-dimensional image data at a pixel coordinate in the two-dimensional image data corresponding to the first coordinate.

CLAIM OF PRIORITY

This application is a continuation of U.S. patent application Ser. No.15/367,221, filed Dec. 2, 2016.

This application claims priority to U.S. Provisional Application No.62/261,932, which is entitled “Multi-Wavelength Depth Encoding Methodfor 3D Range Geometry Compression,” and was filed on Dec. 2, 2015, theentire contents of which are hereby incorporated herein by reference.

GOVERNMENT INTEREST

This invention was made with government support under CMMI-1300376awarded by the National Science Foundation. The government has certainrights in the invention.

TECHNICAL FIELD

This disclosure relates generally to the field of image processing and,more specifically, to methods and systems for encoding and decodinggeometry data corresponding to physical objects in three-dimensionalimaging.

BACKGROUND

Recent advances in 3D scanning technologies have brought about thecapabilities to capture high-quality data at very fast speeds. Givensuch progress, one might consider these technologies to be on the brinkof widespread dissemination. One inherent problem that must be furtheraddressed, however, is establishing methods for 3D range datacompression that are robust and offer high compression ratios; suchmethods will ensure efficient storage and fast, high-quality datarecovery.

Currently, one conventional storage standard for a single frame of 3Dgeometry is the mesh format. These formats (e.g. OBJ, PLY, STL) aregeneric in nature and perform their tasks well, storing the coordinatesof each vertex often along with connectivity information. Additionalinformation can also be stored with the mesh such as a surface normalmap and a (u, v) map. Although these formats are able to perform theirtask of representing a frame of 3D geometry, they also require a largeamount of storage to do so. For example, a single 640×480 frame of 3Dgeometry, with only vertex locations and connectivity information, needsabout 13 MB of space. For real-time or faster 3D capture systems thatwant to store or stream each single frame, this large file size becomesan issue.

Given this, other 3D range data compression techniques have beenproposed. One such methodology is to encode the raw 3D data in some waysuch that it can be represented within a 2D image. The informationstored within an image's color channels can then be used to recover andreconstruct the compressed geometry. Such approaches are able to takeadvantage of very well established image formats (e.g., PNG) and theinfrastructure built around them.

To compress 3D geometry into a 2D image, one approach is to use theprinciples of virtual digital fringe projection (DFP). Using theconventions of DFP along with a virtual structured light scanner, thisHoloImage approach projects fringe images upon the virtual 3D geometryand captures them virtually. The resulting captured fringe images arethen packed into the image (e.g., into its color channels) along withany information that may be required to unwrap the phase map between thephase images (e.g., stair image). An additional advantage to using avirtual fringe projection system which converts raw 3D geometry into a2D image frame is its portability to video storage and streaming.

While digital fringe projection techniques that are known to the art canbe useful for generation of the representations of somethree-dimensional structures based on 2D images, the existing DFPtechniques still have noticeable problems with common lossy imagecompression algorithms such as JPEG compression. While lossy compressionalgorithms by their very nature introduce some errors into compressedimages, the artifacts that present little or no image qualitydegradation in traditional compressed photographs often introduceunacceptably large errors when applied to two-dimensional images thatcontain encoded DFP data. This makes recording of high-resolutionfeatures of three-dimensional objects difficult because the common andhighly effective lossy compression algorithms often produce errors thatrender high-resolution DFP data unusable in practical systems.Consequently, improvements to processes for encoding and decoding DFPdata that improve the quality of the DFP for high-resolution details ofan object and that maintain high quality even when heavily compressedwould be beneficial to the art.

SUMMARY

This disclosure presents a novel method for representingthree-dimensional (3D) range data within regular two-dimensional (2D)images using multi-wavelength encoding. These 2D images can then befurther compressed using traditional lossless (e.g., PNG) or lossy(e.g., JPEG) image compression techniques. Current 3D range datacompression methods require significant filtering to reduce lossycompression artifacts. The nature of the proposed encoding, however,offers a significant level of robustness to such artifacts brought aboutby high levels of JPEG compression. This enables extremely highcompression ratios while maintaining a very low reconstruction errorpercentage with little to no filtering required to remove compressionartifacts. For example, when encoding 3D geometry with the proposedmethod and storing the resulting 2D image using a commercially availableJPEG image compression engine, compression ratios of approximately 935:1versus the OBJ format can be achieved at an error rate of approximately0.027% without any filtering.

In one embodiment, a computer readable medium contains computerinstructions that, when executed by processor, are configured togenerate encoded depth map data for a physical object based on digitalfringe projection (DFP) data has been developed. This includes,receiving, with a processor, digital fringe projection datacorresponding to a three-dimensional structure of a physical object,generating, with the processor, a first fringe encoding and a secondfringe encoding for a first predetermined wavelength based on the DFPdata at a first coordinate, generating, with the processor, a thirdfringe encoding and a fourth fringe encoding for a second wavelengthbased on the DFP data at the first coordinate, the second wavelengthbeing longer than the first wavelength, generating, with the processor,a combined fringe encoding based on the third fringe encoding and thefourth fringe encoding, and storing, with the processor, the firstfringe encoding data, the second fringe encoding data, and the combinedfringe encoding data in a pixel of two-dimensional image data at a pixelcoordinate in the two-dimensional image data corresponding to the firstcoordinate, the two-dimensional image data being stored in a memory.

In another embodiment, a computer readable medium contains computerinstructions that, when executed by processor are configured to performa method for decoding depth map data for a physical object from encodeddepth map data that correspond to DFP data for the physical object hasbeen developed. The method includes retrieving, with a processor, afirst pixel of two-dimensional image data from a memory, decoding, withthe processor, dense phase data stored in the first pixel, decoding,with the processor, wrapped phase data stored in the first pixel,generating, with the processor, a stair image including a second pixelcorresponding to the first pixel of the two-dimensional image data basedon the dense phase data and the wrapped phase data, and generating, withthe processor, a depth map including a third pixel corresponding to thesecond pixel of the stair image, the third pixel storing datacorresponding to a depth of a location of the physical object.

In another embodiment, a system configured to generate encoded depth mapdata for a physical object based on DFP data has been developed. Thesystem includes a memory and a processor operatively connected to thememory. The processor is configured to receive digital fringe projection(DFP) data corresponding to a three-dimensional structure of a physicalobject, generate with the processor, a first fringe encoding and asecond fringe encoding for a first predetermined wavelength based on theDFP data at a first coordinate, generate a third fringe encoding and afourth fringe encoding for a second wavelength based on the DFP data atthe first coordinate, the second wavelength being longer than the firstwavelength, generate a combined fringe encoding based on the thirdfringe encoding and the fourth fringe encoding; and store the firstfringe encoding data, the second fringe encoding data, and the combinedfringe encoding data in a pixel of two-dimensional image data at a pixelcoordinate in the two-dimensional image data corresponding to the firstcoordinate, the two-dimensional image data being stored in the memory.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a system that generates virtual digitalfringe projection (DFP) data corresponding to the three-dimensionalstructure of a physical object.

FIG. 2A is a block diagram of a process for multi-wavelength encoding ofDFP data into a two-dimensional image data format.

FIG. 2B is a block diagram of a process for decoding themulti-wavelength encoded data that are generated during the process ofFIG. 2A.

FIG. 3 is a diagram depicting a process for recording reflections ofstructured sinusoidal light patterns that are projected onto the surfaceof an object.

FIG. 4 is a diagram depicting a set of fringe patterns and other imagedata that are generated during the processes of FIG. 2A and FIG. 2B.

FIG. 5 is a diagram depicting image quality of depth maps that aregenerated as two-dimensional images with prior art techniques and theprocesses of FIG. 2A and FIG. 2B.

FIG. 6 is another diagram depicting image quality of depth maps that aregenerated as two-dimensional images with prior art techniques and theprocesses of FIG. 2A and FIG. 2B.

FIG. 7 is a diagram depicting errors generated in a phase map of a priorart direct depth encoding and decoding process when generating depthdata for a high-resolution surface compared to a phase map generated bythe processes of FIG. 2A and FIG. 2B.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of theembodiments disclosed herein, reference is now be made to the drawingsand descriptions in the following written specification. No limitationto the scope of the subject matter is intended by the references. Thepresent disclosure also includes any alterations and modifications tothe illustrated embodiments and includes further applications of theprinciples of the disclosed embodiments as would normally occur to oneskilled in the art to which this disclosure pertains.

FIG. 1 depicts a system 100 that generates depth data for an objectusing a digital fringe projection (DFP) process. The system 100 includesa camera 104, a projector 108, a display or other output device 112, aprocessor 128, and a memory 132. In the illustrative example of FIG. 1,the system 100 generates imaging data of an object 180 including adigital fringe projection (DFP) representation of the object 180. Thecamera 104 is, for example, a digital camera that captures lightreflected from the surface of the object 180 at a plurality ofwavelengths including for example, all or a portion of the visible lightspectrum, infrared, and ultraviolet frequency ranges. The projector 180generates a series of specifically configured light patterns that areprojected onto the surface of the object 180. The light patternsinclude, for example, a series of predetermined fringe patterns thatcorrespond to alternating regions of high intensity and low intensitylight in a sinusoidal pattern. The predetermined light pattern is alsoreferred to as a “structured” light pattern. The object 180 reflects thepredetermined phase pattern of light, and the three-dimensionalstructure of the object 180 affects how the phase pattern reflects andis captured by the camera 104. FIG. 3 depicts a sinusoidal phase pattern304 that a projector, such as the projector 108 of FIG. 1, projects ontothe surface of an object. In FIG. 3 the structure of the object affectsthe reflected light from the phase pattern 308. The camera 104 generatesa digital image 312 of the reflection of the structured light, which isalso referred to as a “digital fringe pattern”.

In the system 100, the processor 128 is, for example, a digitalprocessing device including one or more central processing unit (CPU)and optionally one or more graphics processing unit (GPU) cores, adigital signal processor (DSP), field programmable gate array (FPGA), orapplication specific integrated circuit (ASIC) that implement an imageprocessing system. In particular, the processor 128 is configured togenerate two-dimensional image data with encoded virtual digital fringeprojection (DFP) data that correspond to the three-dimensional structureof an object, such as the object 180 of FIG. 1. The processor 128 oranother digital processing device is further configured to decode theencoded DFP data to enable the system 100 or another computing device toanalyze the structure of the object 180 or other physical objects withstructural information that is encoded using the systems and methodsdescribed herein.

In the system 100 the memory 132 further includes one or more digitaldata storage devices including, for example, a non-volatile data storagedevices such as a magnetic, optical, or solid state drive and volatilememory including static or dynamic random access memory (RAM), and thedigital storage devices in the memory 132 form a non-transitory computerreadable medium. In the system 100, the memory 132 holds stored programinstructions 136 that enable the system 100 to generate DFP data, encodethe DFP data as two-dimensional image data, and decode thetwo-dimensional image data to retrieve three-dimensional structuralinformation about the object 180. The memory 132 also stores virtual DFPdata as digital data 138 that is generated based on a set of data thatthe camera 104 generates for multiple images that correspond toreflections of the structured light patterns from the surface of theobject 180 as the projector 108 emits the structured light patterns withdifferent phases during operation of the system 100. Virtual DFP data isfurther generated based on a limited set of recorded images of thestructured light in different phases (e.g. three images for threephases) that the processor 128 or another processor generates using apredetermined model with a “virtual” version of the camera 104 andprojector 108 in a virtual three-dimensional environment. The virtualDFP technique reduces the number of structured light images that need tobe encoded to produce a depth map, and the generation of the virtual DFPdata prior to the encoding and decoding techniques that are discussedbelow is otherwise known to the art. The memory 132 further storesencoded image data 140 that includes two-dimensional images with encodeddepth information based on the DFP data 138. The encoded image data 140are optionally compressed using prior-art two-dimensional imagecompression techniques including both lossy and lossless compressiontechniques.

In the system 100, the display or output devices 112 include, forexample, two-dimensional or three-dimensional display devices thatproduce an output corresponding to DFP data that the system 100generates based on various physical objects including the object 180. Inanother configuration, the output device 112 is, for example, a networkinterface device that transmits encoded DFP data from the system 100 toexternal computing devices with the data being compressed to providehigh quality DFP models that consume fewer storage and network bandwidthresources than prior art DFP encoding techniques with comparablequality. The external computing devices then decode the DFP data for awide range of applications including, but not limited to, computer aideddesign and modeling (CAD/CAM) applications, building and vehicleinspections, forensic sciences, entertainment, 3D telecommunications,virtual and augmented reality applications, and the like.

During operation, the system 100 generates three-dimensional informationabout the object 180 using a phase-shifting technique based on thepredetermined structure of the light that the projector 108 emits andthe recorded reflected light that the camera 104 captures duringoperation of the system 100. The projector 108 translates the structuredlight image across the surface of the object 180 to enable the camera104 to generate a series of digital fringe pattern images with adifferent phase for the structured light for a given location of theobject in each image. For example, a single location on the object 180receives the peak of sinusoid for the structured light in one phase, anda trough of the sinusoid for the light in another phase. For example, inone embodiment the system 100 generates three different phase-shiftedimages with predetermined phase shifts (e.g. 2π/3 radians between eachphase), although alternative configurations can employ a differentnumber of phases and phase offsets.

The processor 128 generates a two-dimensional phase map that encodesthree-dimensional information about the object 180, including the depthof different locations on the object 180 as a distance from the camera104, based on the predetermined locations of the camera 104, projector108, and the object 180 in addition to the predetermined structuredlight patterns that the system 100 projects onto the object 180 and thereflected light that the camera 104 receives from the object 180. Thecaptured data is also referred to as a virtual digital fringe projectionprocess because the processor 128 generates the three-dimensionalinformation corresponding to the shape of the object 180 based on avirtual software model using only the series of two-dimensional imagesgenerated by the camera 104 for the reflected light from the object 180from the structured light patterns in the predetermined phases. Theimaging system 100 stores the three-dimensional position data for eachlocation identified on the object 180 as a two-dimensional image thatincludes a two-dimensional array of pixels. To representthree-dimensions in a two-dimensional image, two axes (x and y) for eachlocation on an object are encoded based on the location of acorresponding pixel in the two-dimensional image. The depth dimension(z) is encoded in the data contents of the pixel itself. For a digitalfringe image, the depth is not merely a simple scalar value but acombination of phase information for the multiple structured lightphases.

While certain techniques for generation of DFP data representing depthinformation for a three-dimensional object are known to the art, thesystem 100 is configured to generate an encoding for the fringe imagesfrom the DFP process with a multi-wavelength encoding process and acorresponding decoding process that are not known to the art. The system100 enables storage of the encoded data to encode the three-dimensionaldepth information about an object, such as the object 180, in atwo-dimensional set of image data with a higher level of quality for agiven size of data required to encode the depth information compared toprior art systems.

FIG. 2A depicts a process 200 for multi-wavelength encoding of depthinformation data that are generated in a digital fringe projectionprocess. In the description of process 200, a reference to the processperforming a function or task refers to the execution of stored programinstructions by a processor in an image processing system to perform thefunction or action. The process 200 is described in conjunction with thesystem 100 of FIG. 1 for illustrative purposes.

Process 200 begins as the processor 128 receives virtual digital fringeprojection (DFP) data that are generated for a physical object (block204). As described above, the system 100 or another suitable digitalimaging system generates the DFP data. In the system 100, the processor128 retrieves the virtual DFP data 138 from the memory 132, although inalternative embodiments a processor receives the DFP data via a datanetwork or other communication channel. As described above, the DFP datainclude a two-dimensional array of elements that correspond to imagedata that the camera 104 generates in response to receiving reflectedstructured light patterns from the surface of the object 180.

The process 200 continues as the processor 128 generates first andsecond fringe encodings based on the DFP data at a predeterminedwavelength P corresponding to a predetermined fringe width (block 208).In one configuration, the selected wavelength is set as a fraction (e.g.¼ or ⅕) of a maximum wavelength that is included in the DFP data, wherethe term “wavelength” here refers to the widths of the fringes that areincluded in the DFP data. More generally, the first wavelength P isselected to be shorter than the maximum wavelength in the DFP data. Themaximum wavelength in a two-dimensional depth map Z corresponds to afull range of depths in the DFP data corresponding to the change indepth in the structure of the object 180. For example, in FIG. 1 if theclosest point on the object 180 is at a distance of 10 cm from thecamera 104 and the farthest point is at a distance of 20 cm, then themaximum range of Z and corresponding maximum wavelength is 20 cm−10cm=10 cm for the object 180. In the system 100, the processor 128generates two encodings for the DFP data based on sine and cosinefunctions as is set forth below:

${I_{r}\left( {i,j} \right)} = {0.{5\left\lbrack {1.0 + {\sin\ \left( \frac{2\pi \times {Z\left( {i,j} \right)}}{P} \right)}} \right\rbrack}}$${I_{b}\left( {i,j} \right)} = {0.{5\left\lbrack {1.0 + {\cos\ \left( \frac{2\pi \times {Z\left( {i,j} \right)}}{P} \right)}} \right\rbrack}}$

In the equations listed above, i and j represent coordinates for one setof phase data in a two-dimensional arrangement of the DFP data, and theprocessor 128 generates the sine and cosine encodings for eachcoordinate in the DFP data. Z represents a two-dimensional array of theDFP data prior to encoding, such as the DFP data 138 stored in thememory 132 of the system 100. In the embodiment of FIG. 2A thetwo-dimensional image format encodes a two-dimensional array of pixelswhere each pixel includes numeric data corresponding to three differentcolor channels, with red, green, and blue (RGB) color channels being onecommon structure in many existing two-dimensional image data formats.The processor 128 places each encoded value I_(r) in the red colorchannel of a pixel in a two-dimensional RGB encoded image and each I_(b)encoded value in the blue color channel of the corresponding pixel ofthe RGB formatted image. However, alternative embodiments of the process200 can store the encoded data in different arrangements of the colorchannels and other consistent selections of color channels arecompatible with different embodiments of the process 200.

The process 200 continues as the processor 128 generates third andfourth fringe encodings for the DFP data (block 212). The third andfourth encodings are based on the longest wavelength Z and the range ofwavelengths in the DFP data, which is referred to as Range(Z), whichrepresents the difference between the largest and smallest values withinthe DFP data Z. During the process 200 the third encoding is based on asine function and the fourth encoding is based on a cosine function. Theprocessor 128 generates the third and fourth encodings based on thefollowing functions:

${I_{g}^{sin}\left( {i,j} \right)} = {0.{5\left\lbrack {{1.0} + {\sin\left( {{2{\pi\left( \frac{{Mod}\left( {{Z\left( {i,j} \right)},{{Range}(Z)}} \right.}{{Range}(Z)} \right)}{I_{g}^{cos}\left( {i,j} \right)}} = {0.{5\left\lbrack {{1.0} + {\cos\left( {2{\pi\left( \frac{{Mod}\left( {{Z\left( {i,j} \right)},{{Range}(Z)}} \right.}{{Range}(Z)} \right)}} \right.}} \right.}}} \right.}} \right.}}$

In the equations above, the sine and cosine encodings I_(g) ^(sin) andI_(g) ^(cos) each generate encoded data for a pixel in a two-dimensionalarray of image data that corresponds to the individual DFP data elementat coordinates (i,j). The Mod function refers to a numeric modulooperation. To store the different third and fourth encoding values in asingle color channel of a two-dimensional image format, the processor128 generates a combined encoding value based on the third and fourthencoded values (block 216). In the embodiment of FIG. 2A, the processor128 generates the combined encoding value based on the followingfunction:

${\phi_{g}\left( {i,j} \right)} = {\tan^{- 1}\left( \frac{{I_{g}^{\sin}\left( {i,j} \right)} - 0.5}{{I_{g}^{\cos}\left( {i,j} \right)} - 0.5} \right)}$

The function ϕ_(g)(i,j) generates values in a numeric range of (−π, π],which may not be compatible with the numeric encoding conventions ofsome two-dimensional image formats. The processor 128 normalizes thenumeric range of the results from ϕ_(g)(i,j) to another numeric range,such as (0, 1], for a color channel of a two-dimensional image formatusing the following function:

${I_{g}\left( {i,j} \right)} = \frac{{\phi_{g}\left( {i,j} \right)} + \pi}{2\pi}$

During the process 200, the processor 128 generates encoded pixels inthe two-dimensional image corresponding to each coordinate (i, j) in theDFP data using the three color channels of each pixel in thetwo-dimensional image to store the three encoded values corresponding tothe first encoded value, the second encoded value, and the combinationof the third and fourth encoded values (block 220). As described, above,the process 200 is referred to as a multi-wavelength DFP encodingprocess because the first and second fringe encodings are generatedbased on the shorter wavelength P while the combined fringe encodingdata are generated based on the longer wavelength Z. Thus, the process200 generates a two-dimensional image in which the depth information forthe DFP data at a particular location on an object are encoded based onthe multiple wavelengths in multiple color channels of a pixel in atwo-dimensional image. In the system 100, the processor 128 performs theoperations described above with reference to blocks 208-220 to generatethe encoded values for the color channels in the two-dimensional imagein any order or concurrently.

During the process 200, the processor 128 optionally compresses thetwo-dimensional encoded image data using a lossy or lossless compressionprocess (block 224). For example, in one configuration the processor 128uses a JPEG compression process that compresses the two-dimensionalimage data into a lossy compression format that reduces the accuracy ofthe compressed image to some degree with the benefit of greatly reducingthe size of the stored two-dimensional data. In another configuration,the processor 128 applies a lossless compression process to generate alossless compressed image format using, for example, the portablenetwork graphics (PNG) format. The lossless compression process retainsall of the information from the originally encoded information but insome instances the lossless compression process produces a compressedimage that requires more memory storage capacity than the lossycompression formats. The processor 128 stores the encoded and optionallycompressed image data 140 in the memory 132. In some embodiments, theprocessor 128 transmits the encoded data to another computing systemusing a network interface device 112.

FIG. 2B depicts a decoding process 250 that enables decoding of theencoded depth information data that the system 100 generates during theprocess 200. In the description of process 250, a reference to theprocess performing a function or task refers to the execution of storedprogram instructions by a processor in an image processing system toperform the function or action. The process 250 is described inconjunction with the system 100 of FIG. 1 for illustrative purposes.

The process 250 begins as the processor 128 retrieves data that arestored in the two-dimensional image from the memory (block 254). In thesystem 100, the processor 128 retrieves the encoded image data 140 fromthe memory 132. In embodiments where the encoded image data are storedin a compressed format, the processor 128 also performs a decompressionprocess to retrieve the two-dimensional image data with multiple colorchannels that store the encoded DFP information.

Process 250 continues as the processor 128 decodes a dense phase mapfrom the encoded DFP data corresponding to the first and second fringeencodings that are stored in two color channels (e.g. the red and bluecolor channels) of the two-dimensional image data (block 258). In thesystem 100, the processor 128 initiates decoding of the retrieved DFPdata using the following arctangent function:

${\phi_{rb}\left( {i,j} \right)} = {\tan^{- 1}\left( \frac{{I_{r}\left( {i,j} \right)} - 0.5}{{I_{b}\left( {i,j} \right)} - 0.5} \right)}$

In the function above, I_(r) (i,j) represents the red-channel componentdata in a pixel at coordinates (i,j) that corresponds to the firstencoded DFP data generated during the process 200. Similarly, I_(b)(i,j) represents the blue-channel component data in a pixel atcoordinates (i,j) that corresponds to the second encoded DFP datagenerated during the process 200, although as mentioned above theprecise color channel assignments for each set of encoded DFP data mayvary between embodiments of the processes 200 and 250. The elementϕ_(rb) is also referred to as a “dense” phase map since this element isbased on the first and second encoded elements from the original DFPdata. The processor 128 performs the same function for each of thepixels in the two-dimensional image data to generate a two-dimensionaldense phase map based on the image data.

During the process 250, the processor 128 also decodes a wrapped phasemap from the color channel of each pixel in the image data that storesthe combined encoded DFP data, such as the green color channel in theillustrative embodiments of FIG. 2A and FIG. 2B (block 262). Theprocessor 128 retrieves the wrapped phase map data using the followingfunction:ϕ_(g)(i,j)=I _(g)(i,j)(2π)−π

In the function above, I_(g) (i,j) is the green channel value for apixel at coordinates i,j in the two-dimensional image. The equationabove also includes the multiplication by 2π and subtraction by π toremap the numeric values that the processor 128 retrieves from the imagedata from a range of (0, 1] to a range of phases in a numeric range of(−π, π]. The processor 128 performs the same function for each of thepixels in the two-dimensional image data to generate a two-dimensionalwrapped phase map based on the image data.

The process 250 continues as the processor 128 generates a “stair”image, which is a two-dimensional image with pixel values that aregenerated based on both the extracted dense phase map and wrapped phasemap data (block 266). The “stair” image is so named because the imageincludes a finite number of depth levels for different pixels just as astaircase has a finite number of steps, although of course thearrangement of pixels in the image represent the structures of objectswith a wide range of shapes. The processor 128 generates the stair imageusing the following function:

${K\left( {i,j} \right)} = \frac{\left( {{\phi_{g}\left( {i,j} \right)} \times \frac{{Range}(Z)}{P}} \right) - {\phi_{rb}\left( {i,j} \right)}}{2\pi}$

In the equation above, K(i,j) represents the value of a pixel in thestair image at coordinates (i, j) based on the corresponding wrappedphase map value ϕ_(g)(i,j) and dense phase map value ϕ_(rb)(i,j) for thesame coordinates. The Range(Z) and P values are the same maximum Zwavelength range and predetermined shorter wavelength values,respectively, that were used in the encoding process 200 that generatedthe encoded DFP data.

In the process 250, the processor 128 unwraps the dense phase map usingthe stair image to obtain an unwrapped phase map (block 268). Theunwrapping process uses each element of K(i, j) in the stair image todetermine the scale of how many phase periods (represented numericallyas 2π radians per period) must be added to each corresponding elementϕ_(rb)(i,j) in the dense phase map to remove the 2π discontinuities fromthe stored dense phase map data. Removing the phase jumps produces acontinuous, unwrapped phase map, Φ, of the shorter wavelength P.

The process 250 continues as the processor 128 generates an output depthmap based on the unwrapped phase map (block 270). The output depth mapis another two-dimensional arrangement of pixels in which thecoordinates (i, j) correspond to the x and y axis locations,respectively, of different features on the object and the numeric valueof each element in the depth map corresponds to the z-axis depthlocation of the surface of the object at each location. The processor128 generates the output depth map Z as a two-dimensional image thatcorresponds to the three-dimensional structure of the object as viewedfrom one viewing position using the following functions, where Φ(i, j)is the unwrapped phase map described above and Round represents anumeric rounding function:

Φ(i, j) = ϕ_(rb)(i, j) + 2π × Round(K(i, j))${Z\left( {i,j} \right)} = \frac{{\Phi\left( {i,i} \right)} \times P}{2\pi}$

As described above, the system 100 or another suitable computing devicegenerates output based on the depth map to enable a wide range of usefulapplications. Examples of applications that use to the structure of thedepth map include, but are not limited to, computer aided design andmodeling (CAD/CAM) applications, building and vehicle inspections,forensic sciences, entertainment, 3D video telepresence, virtual andaugmented reality applications, and the like. Because most of 3D imagingdevices inherently use 2D sensor to capture 2D images and recover depthinformation from 2D images, and thus the depth information is criticalfor the majority of 3D imaging technologies.

FIG. 4-FIG. 9 depict elements of the processes 200 and 250 that aredescribed above. FIG. 4 depicts shows each channel and step along themulti-wavelength encoding and decoding processes. FIG. 4 shows thecompressed, encoded 2D color image (a). In FIG. 4, images (b)-(d) depictthe individual red, green, and blue color channels of the image,respectively. To recover geometry from encoded image data, the processor128 uses the dense phase corresponding to the fringe images of theshorter wavelength, and FIG. 4 depicts the resulting dense phase image(e). As described previously, the continuous, wrapped phase of thelonger wavelength as depicted in image (c) of FIG. 4 forms a basis togenerate the stair image (f) of FIG. 4. As described above, theprocessor uses the stair image data to unwrap the phase of the shorterwavelength to generate the unwrapped phase map (g) in FIG. 4, which inturn forms the basis for the depth map (h) that corresponds to thethree-dimensional structure of the object.

FIG. 5 depicts a visual comparison of the decoded depth information fora three-dimensional sphere object based on the processes 200 and 250 ina system that uses a lossy JPEG compression format to compress thetwo-dimensional image data generated during the process 200. Each columnin FIG. 5 Each column represents JPEG storage levels 100%, 80%, 60%,40%, and 20%, respectively, using a commercially available JPEGcompression engine. Row 1 (items a-e) depicts the results of a prior artdirect depth method without a filter applied to the recovered geometry;Row 2 (items f-j) depicts the prior art direct depth method with a 25×25filter applied to the recovered geometry; and Row 3 (items k-o) depictsthe multi-wavelength depth techniques that are taught herein without anyfilter applied. As depicted in FIG. 5, even without the use of a filterthe techniques taught herein produce high quality results. Inparticular, the multi-wavelength encoding and decoding techniques taughtherein produce high quality results in configurations that employ highlevels of lossy compression where the prior art techniques experiencemuch larger degradations in image quality.

Table 1 presents a comparison of the quality of results of themulti-wavelength encoding/decoding (MWD) processes of FIG. 2A and FIG.2B compared to two versions of a prior-art “direct depth” encodingprocess (DD) in a system that uses a lossless compression format such asportable network graphics (PNG) or a lossy compression format such as aJPEG compression format at varying quality levels from highest quality(100) to lowest quality (20). The quality metric in Table 1 is thepercentage of root mean square (RMS) error between the original encodeddata and the decoded data for the sphere structure that is depicted inFIG. 5:

TABLE 1 PNG JPG100 JPG80 JPG60 JPG40 JPG20 (%) (%) (%) (%) (%) (%) DD NoFilter 0.0061 2.9557 3.4834 3.3970 3.8545 4.5275 DD 25 × 25 0.00610.0263 0.0438 0.0578 0.0725 0.0924 Filter MWD No 0.0061 0.0167 0.02710.0508 0.0651 0.0928 Filter

As depicted in Table 1, the MWD processes that are taught in thisdocument produce quality levels that are equivalent to the prior art inlossless compression situations (PNG) and that are comparable to orsignificantly better than the prior art in the lossy JPEG scenarios. Inparticular, while the prior art DD process with the use of an additional25×25 filter sometimes produces results with similar accuracy levels tothe MWD processes taught herein, the MWD processes do not require theuse of additional filters to achieve the high quality results, whichreduces the complexity of the encoding and decoding MWD processescompared to the prior art.

In some instances, a small filter can be used to further improve theresults of the processes 200 and 250 when using lossy compression. FIG.6 depicts an example of errors in 3D image data that are generated usingprior art techniques (drawings (a) and (b)) compared to the MWDprocesses that are described herein (drawing (c)). FIG. 6 depicts therecovered data for a hemispherical three-dimensional object. Drawing (a)depicts the prior art (a) 3D recovery from an image generated using thedirect depth (DD) process and a JPEG 10% compressed image with a small5×5 filter. Drawing (b) depicts the same object recovered from an imagegenerated using the DD process and the JPEG 10% compressed image with alarger 25×25 filter. Drawing (c), however, depicts the results from theMWD encoding and decoding processes of FIG. 2A and FIG. 2B us with theJPEG 10% compression level and only a small 5×5 filter. As seenqualitatively, the hemisphere in drawing (c) is much more accurate thanin drawings (a) and (b). Quantitatively, drawings (a) and (b) generatedusing the prior-art techniques produce RMS errors of 6.13% and 3.93%,respectively, while drawing (c) that corresponds to an image generatedusing the processes described herein has an RMS error of only 0.15%.

FIG. 7 depicts another example of the inability of the prior art DDtechnique to generate accurate depictions of high-resolution smoothtransitions in depth for the structure of a three-dimensional object,while the MWD techniques taught herein enable encoding and decoding ofhigh-resolution depth data. In FIG. 7, diagram (a) is a normalizedsubwindow of the recovered phase when using the prior art direct depthtechnique to encode geometry with a depth resolution of 1,024 and 128stairs. Diagram (b) is a cross section of the recovered phase from (a)with the slope removed and the resulting 2π-tall spikes leave artifactswithin the data for the prior art direct depth technique. In FIG. 7,diagram (c) depicts the normalized subwindow of recovered phase whenusing the MWD technique described herein to encode the same geometrywith the same number of stairs; (d) cross section of the recovered phase(c) with the slope removed, which clearly produces a more accuraterepresentation of the smooth depth transition of the three-dimensionalobject.

It will be appreciated that variants of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be desirablycombined into many other different systems, applications or methods.Various presently unforeseen or unanticipated alternatives,modifications, variations or improvements may be subsequently made bythose skilled in the art that are also intended to be encompassed by thefollowing claims.

What is claimed is:
 1. A non-transient computer readable mediumcontaining computer executable instructions which are configured to,when executed by a processor: receive digital fringe projection (DFP)data corresponding to a three-dimensional structure of a physicalobject; generate a first fringe encoding and a second fringe encodingfor a first predetermined wavelength based on the DFP data at a firstcoordinate; generate a third fringe encoding and a fourth fringeencoding for a second wavelength based on the DFP data at the firstcoordinate, the second wavelength being longer than the firstwavelength; generate a combined fringe encoding based on the thirdfringe encoding and the fourth fringe encoding; and store the firstfringe encoding data, the second fringe encoding data, and the combinedfringe encoding data in a pixel of two-dimensional image data at a pixelcoordinate in the two-dimensional image data corresponding to the firstcoordinate, the two-dimensional image data being stored in a memory. 2.The computer readable medium of claim 1, wherein the computer executableinstructions are further configured to, when executed by the processor:store the first fringe encoding data in a first color channel of thepixel, the second fringe encoding data in a second color channel of thepixel, and the combined fringe encoding data in a third color channel ofthe pixel.
 3. The computer readable medium of claim 1, wherein thecomputer executable instructions are further configured to, whenexecuted by the processor: compress the two-dimensional image data priorto storing the two-dimensional image data in the memory.
 4. The computerreadable medium of claim 3, wherein the computer executable instructionsare further configured to, when executed by the processor: compress thetwo-dimensional image data using a lossy compression format.
 5. Thecomputer readable medium of claim 3, wherein the computer executableinstructions are further configured to, when executed by the processor:compress the two-dimensional image data using a lossless compressionformat.
 6. The computer readable medium of claim 1, wherein the computerexecutable instructions are further configured to, when executed by theprocessor: generate the first fringe encoding based, at least in part,on a sine function value of a proportion of the second wavelengthdivided by the first predetermined wavelength; and generate the secondfringe encoding based, at least in part, on a cosine function value ofthe proportion of the second wavelength divided by the firstpredetermined wavelength.
 7. The computer readable medium of claim 1,wherein the computer executable instructions are further configured to,when executed by the processor: generate the third fringe encodingbased, at least in part, on a sine function value of a proportion of thesecond wavelength divided by a maximum range of wavelengths present inthe DFP data; and generate the fourth fringe encoding based, at least inpart, on a cosine function value of the proportion of the secondwavelength divided by a maximum range of wavelengths present in the DFPdata.
 8. The computer readable medium of claim 7, wherein the computerexecutable instructions are further configured to, when executed by theprocessor: generate, with the processor, the combined fringe encodingbased, at least in part, on an arctangent of a proportion of the thirdfringe encoding divided by the fourth fringe encoding.
 9. Anon-transient computer readable medium containing computer executableinstructions which, when executed by a processor, decode encoded digitalfringe projection (DFP) data corresponding to a physical object by:retrieving a first pixel of two-dimensional image data from a memory;decoding a plurality of phase data stored in a plurality of colorchannels of the first pixel; generating a stair image including a secondpixel corresponding to the first pixel of the two-dimensional image databased on the plurality of phase data; and generating a depth mapincluding a third pixel corresponding to the second pixel of the stairimage, the third pixel storing data corresponding to a depth of alocation of the physical object.
 10. The computer readable medium ofclaim 9, wherein the computer executable instructions are furtherconfigured to, when executed by the processor, decode the encoded DFPdata corresponding to a physical object wherein the plurality of phasedata comprises dense phase data and wrapped phase data.
 11. The computerreadable medium of claim 10, wherein the computer executableinstructions are further configured to, when executed by the processor,decode the dense phase data based, at least in part, on an arctangent ofa proportion of a first fringe encoding stored in a first color channelof the first pixel divided by a second fringe encoding stored in asecond color channel of the first pixel.
 12. The computer readablemedium of claim 10, wherein the computer executable instructions arefurther configured to, when executed by the processor, retrieve thewrapped phase data from a third color channel of the first pixel; andscale the wrapped phase data from a first numeric scale to a secondnumeric scale with a range of values from [−π, π).
 13. The computerreadable medium of claim 10, wherein the computer executableinstructions are further configured to, when executed by the processor,generate the second pixel corresponding to the first pixel of thetwo-dimensional image data based, at least in part, on the wrapped phasedata subtracted from a product of the dense phase data multiplied by aproportion of a maximum range of wavelengths present in the encoded DFPdata divided by a predetermined wavelength that was used to generate theencoded dense phase data stored in the pixel, the predeterminedwavelength being smaller than the maximum range of wavelengths.
 14. Thecomputer readable medium of claim 13, wherein the computer executableinstructions are further configured to, when executed by the processor,generate the third pixel corresponding to the second pixel based, atleast in part, on a sum of the dense phase data added to the secondpixel of the stair image function multiplied by the predeterminedwavelength.
 15. A method for generating encoded depth data comprising:receiving, with a processor, digital fringe projection (DFP) datacorresponding to a three-dimensional structure of a physical object;generating, with the processor, a plurality of fringe encoding valuesbased on the DFP data at a first coordinate; storing, with theprocessor, the plurality of fringe encoding values in a plurality ofcolor channels of a pixel of two-dimensional image data at a pixelcoordinate in the two-dimensional image data corresponding to the firstcoordinate, the two-dimensional image data being stored in a memory. 16.The method of claim 15, wherein generating the plurality of fringeencoding values further comprises: generating a first fringe encodingand a second fringe encoding for a first predetermined wavelength basedon the DFP data at the first coordinate; and generating a third fringeencoding and a fourth fringe encoding for a second wavelength based onthe DFP data at the first coordinate, the second wavelength being longerthan the first wavelength.
 17. The method of claim 16, whereingenerating the plurality of fringe encoding values further comprises:generating a combined fringe encoding based on the third fringe encodingand the fourth fringe encoding.
 18. The method of claim data 16, whereinstoring the plurality of fringe encoding values further comprises:storing the first fringe encoding data in a first color channel of thepixel, the second fringe encoding data in a second color channel of thepixel, and information based on the third encoding data and the fourthencoding value in a third color channel of the pixel.
 19. The method ofclaim 15, the compressing further comprising: compressing, with theprocessor, the two-dimensional image data using a lossy compressionformat.
 20. The method of claim 15, further comprising: compressing,with the processor, the two-dimensional image data using a losslesscompression format.